A computer network typically includes a collection of interconnected computing devices that exchange data and share resources. These devices include, for example, web servers, database servers, file servers, routers, printers, end-user computers and other devices. Each of the variety of devices executes a myriad of different services and communication protocols. Each of the different services and communication protocols exposes the network to different security vulnerabilities.
In certain contexts, such as within a corporate enterprise network, a system administrator or other system owner would like to restrict or limit access to and usage of network resources. For example, the system administrator may wish to prevent network access by an unknown application because the unknown application may be malware, such as a virus, trojan, or other malicious software. The system administrator may also wish to prevent users from using certain applications due to various considerations. For example, the system administrator may wish to block network access to applications such as BitTorrent, Skype, or other “grayware” applications due to consumption of network resources or security concerns with respect to the applications. Therefore, the system administrator may configure a firewall or other network device to block certain applications and protocols.
In order to identify network communications associated with unwanted applications, IDSs or other security systems may analyze packet streams of the network communications and employ behavioral analysis. For example, a security system may perform deep packet inspection and apply patterns to the payloads of the packets in an attempt to identify the source software application. However, developers of applications that are commonly blocked, such as Skype, BitTorrent, and malware such as viruses and trojans, will often employ techniques to circumvent a system administrator's or other user's attempt to block these applications. For example, a virus programmer may code the virus to encrypt all transmissions in accordance with an encryption protocol. For example, the developers may code the application to encrypt an entire payload of a TCP/IP packet, including areas of the packet that would typically be unencrypted. As a result, intrusion detection system (IDS) or other security devices are unable to perform deep packet inspection and often have difficulty identifying the particular type of software application that originated the network communication.
As one example, the Back Orifice protocol depends upon a random number generator that is seeded, i.e. initialized, with a secret key. Both parties to a Back Orifice communication session know the secret key in advance of the communication session, and both parties have the same random number generator. By seeding the random number generator with the same secret key, both parties are able to encrypt data without exchanging a key during the communication session. In this way, the parties seek to avoid detection of any key exchange and session establishment. Moreover, parties to the Back Orifice protocol encrypt all data in the packet, including the Back Orifice application-layer header. Back Orifice uses a 17-byte application-layer header that is encrypted, in addition to the payload, by one party and decrypted by the other party to a communication session.
As another example, various BitTorrent client applications or other peer-to-peer client software applications implement similar encryption schemes in an attempt to avoid detection by security devices. For example, some BitTorrent software clients implement protocol encryption, message stream encryption, or protocol header encryption. Some BitTorrent clients provide an option for users to select between encrypting only the protocol header or the entire packet. To provide encryption, specifically to determine a key for encryption, some BitTorrent clients utilize a distributed hash table (DHT). A DHT includes an infohash, which is the result of a hash function performed on a file. The infohash may be used by each peer of a peer-to-peer file exchange using the BitTorrent protocol such that each peer may mutually generate an encryption key, such as an RC4 encryption key. In this manner, each peer may generate the same encryption key without exchanging the encryption key.
Enterprises may wish to block such applications for a variety of reasons. For example, a system administrator may wish to block the Back Orifice application because a malicious user may intentionally or inadvertently use Back Orifice to install a virus or to take control of a computer or server remotely. As another example, the administrator may wish to block applications that utilize a large amount of bandwidth, such as BitTorrent clients or Skype.
Conventional techniques for identifying software applications associated with encrypted data streams are problematic. For example, a conventional IDS may, for example, attempt to apply behavior analysis to the overall communication session, such as by determining an average size and frequency of data transmission for a certain port or session. If the average size and frequency of data transmission matches known characteristics for a malicious or unwanted application, the IDS may block further communication of that session. However, this method of profiling encrypting communication sessions in an attempt to identify the particular originating software application requires a long series of packet exchanges before detection of the unwanted application. Moreover, this method may fail to identify certain unwanted applications and may trigger false positives for desirable applications.